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TOPOLOGICAL DYNAMICS ON FINITE DIRECTED GRAPHS 


JOSE AYALA AND WOLFGANG KLIEMANN 


Abstract. In this work we establish that finite directed graphs give rise to 
semiflows on the power set of their nodes. We analyze the topological dynamics 
for semiflows on finite directed graphs by characterizing Morse decompositions, 
recurrence behavior and attractor-repeller pairs under weaker assumptions. As 
is expected, the discrete metric plays an important role in our constructions 
and their consequences. The connections between the semiflow, graph theory 
and Markov chains are here explored. We lay the foundation for a dynamical 
systems approach to hybrid systems with Markov chain type perturbations. 


1. Introduction 

The mathematical theory of dynamical systems analyzes, from an axiomatic 
point of view, the common features of many models that describe the behavior of 
systems in time. In its abstract form, a dynamical system is given by a time set T 
(with semigroup operation o), a state space M, and a map $ : T x M M that 
satisfies (i) $(0,a:) = x for all x 6 M, describing the initial value, and (ii) ^{tos,x) = 
$(t,$(s,a;)) for all t,s eT and x e M. At the heart of the theory of dynamical 
systems is the study of systems behavior when t ±oo (qualitative behaviour), 
as well the change in behaviour under variation of parameters (bifurcation theory) 

[iiiniiiiiiiiiTi]. 

In this work we consider dynamical systems on finite directed graphs without 
multiple edges. We analyze their communication structure, i.e., equivalence classes 
of vertices that can be reached mutually via sequences of edges. This leads to the 
set of communicating classes C of a graph and a reachability order < on C. The key 
concept is that of an L-graph corresponding to graphs for which each vertex has 
out-degree > 1. As it turns out, these are exactly the graphs for which the w-limit 
sets of the associated semiflow are nonempty. To each graph G = {V,E), where V 
is the set of vertices, 7^(V) the power set of V, and E c V xV the set of edges, 
we associate a semiflow : N x 'P{V) 'P{V). This semiflow is studied from the 

point of view of qualitative behavior of dynamical systems. We adapt the concepts 
of w-limit sets, (positive) invariance, recurrence, Morse decompositions, attractors 
and attractor-repeller pairs to <I>g and prove characterizations equivalent to those in 
[3]. As it turns out, the finest Morse decomposition of the semiflow f&c corresponds 
to the decomposition of G into the communicating classes C. In addition, the order 
on the communicating classes is equivalent to the order that accompanies a Morse 
decomposition. Moreover, the connected components of the recurrent set of $<3 are 
exactly the (finest) Morse sets of <I>Gj i-e., the communicating classes of G. 
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Graphs G = {V,E) (and certain aspects of Markov chains) are often studied using 
the adjacency matrix Aq- The products of Ac describe the paths, and hence the 
communication structure of G. We construct a semiflow : N x ^ (where 
is the vertex set of the unit cube in that is equivalent to the semiflow 
defined on V{V), using Boolean matrix multiplication. This point of view is some¬ 
what different from the standard approach that uses regular matrix multiplication 
and that does not lead to an equivalent semiflow. The equivalence allows us to 
interpret all results obtained for $g in terms of certain linear iterated function 
systems. 

We subsequently apply the results obtained for graphs and their semiflows to the 
study of general finite Markov chains. Our results are presented in the form of a 
“3-language dictionary”: Each key concept for Markov chains is “translated” into 
graph language and into semiflow language. This dictionary is contained in Facts 
1-13 and Fact 15. Note that our concepts and results in terms of global dynamics 
only deal with the communication structure of graphs (or the qualitative behavior of 
semiflows) and hence they do not contain the probabilistic information of a Markov 
chain. But, it turns out that a simple result on the geometric decay of certain 
probabilities (Lemma 15.101) is sufficient to recapture all the relevant probabilistic 
information. Facts 14 and 16-18 describe the long term behavior of general Markov 
chains and introduce the concept of multistable states. Our presentation unifies 
many related concepts concepts and shows which structural (deterministic, graph 
theoretic, semiflow) properties and which probabilistic properties are really needed 
to analyze Markov chains. Recall that a hybrid system is a dynamical system 
exhibiting the interaction between discrete and continuous phenomena. In this 
work we lay the foundation for a dynamical systems approach to hybrid systems 
with Markov chain type perturbations. 


2. Orbit decomposition of finite directed graphs 

We start by presenting basic definitions and notations used along this work. Our 
first goal is to produce two simple graph decompositions motivated by concepts 
form dynamical systems. In contrast to the standard convention, throughout this 
work, the vertex communication is not in necessarily an equivalence relation. This 
simple observation will lead to interesting phenomena. We define communicating 
sets and classes, and a partial order between communicating classes is presented. 
We introduce a necessary condition to meaningfully study asymptotic behaviour 
via the concept of orbit. Some of the results presented in this section are somehow 
elementary, we prefer to include such results for the sake of exposition. 

Remark 2.1. Throughout this note whenever we refer to a graph G = (V,E) we 
mean a finite directed graph when V is the set of vertices in G and E is the set of 
edges in G such that no multiple edges between any two elements in V are allowed. 

2.1. Orbits and communicating classes. The communication structure in graphs 
is a central concept in this work. In this section we introduce the concepts of com¬ 
municating sets and communicating classes based on the idea of orbits. 

Let G be a graph. An edge from the vertex i to the vertex j is denoted {i,j) ^ E. 
A path in G correspond to a sequence of vertices agree with the incidence and 
direction in G and is denoted by {i^ii ...in)- Sometimes we write i e 7 to specify 
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that the vertex i belongs to the path 7. We define the set: 

(2.1) r" = {7:^(7) = n, neN} 

as the set of all paths 7 of length £{j) = n, with r° = V. We can specify vertices in 
a path 7 in terms of the projection maps for 0 < p < n: 

TTp : r” -> V, TTpi-y) = ip 

where ip is the vertex in 7. In other words, 

A subpath 7' of 7 is a subsequence of 7 of consecutive edges (or vertices) belong¬ 
ing to 7. In particular, any edge of a path is a subpath of length one. Composition 
of paths will play a role in many of our proofs. 

Definition 2 . 2 . For two paths 71 and 72 with ^(71) = m and ^'(72) = n such that 
7i = {i.. .j) and 72 = {j .. .k) we define the concatenation of the paths as 

{71,72) = {i...j)*{j...k) = {i...k) 

with £((i... k)) = m + n and TTm{{i ■ ■ ■ k)) = j. 

Definition 2 . 3 . A vertex i e V has access to a vertex j e V ii there exists a path 
of length > 1 from i to j. We say that the vertices i and j communicate, written 
as f ~ j, if they have mutual access. A subset U of C is a communicating set if any 
two vertices of U communicate. 

Remark 2 . 4 . Note that the relation ~ is reflexive iff for all f e C there exists a 
path ju = {i... i), a property that does not always hold. In general we have that: 
The vertex communication relation ~ is symmetric and transitive but, in general, 
it lacks the reflexivity property. 

We define a smaller set on which reflexivity holds. Denote the union of all 
communicating sets by: 

14 = { * e D : f ~ j for some j eV }. 

For f 6 14 we define [f] = {j eV, j i}. Then I 4 /~= {[z], i € 14 } is a partition of 
14, that is, [z] n [j] = 0 for j i [z], and u[z] = 14- 

Definition 2 . 5 . Let G be a graph with communication relation ~. Each set [z] 
for z € 14 is called a communicating class of G. We denote the set 14 /~ of all 
communicating classes by C. 

Note that by definition, communicating classes are communicating sets. They 
are characterised by their maximality. 

Proposition 2 . 6 . Communicating classes are maximal communicating sets with 
respect to the set inclusion. Vice versa, maximal communicating sets are commu¬ 
nicating classes. 

Proof. Assume that [j] € C is not maximal, then there exist z e [j], and k ^ [j] 
with z ~ A: i.e., [j] is not maximal. Since z e [j] there exist paths 71 = (z.. .j) and 
l2 = (j ■ ■ - i). Moreover, since z and k communicate we have paths 73 = (z... A:) and 
74 = {k...i). The concatenations (74,71) and (72,73) imply k j and therefore 
k € [j], which leads to a contradiction, proving the first claim of the proposition. 
The second part follows by definition of communicating classes. □ 



4 


JOSE AYALA AND WOLFGANG KLIEMANN 


Definition 2 . 7 . The positive and negative orbit of a vertex i are defined as: 
^^(0 = {j s 3n>l, 37€r" such that 7ro(7) = 7r„(7)=j}) 

0~(i) = {j€ V'■ 3n>l, 3 7 € r" such that T^oil) = jj = 

where 7 ro( 7 ) and 7 r„( 7 ) represent the initial and the final vertices in 7 . 

Our first result shows that communicating classes can be characterised using 
orbits of vertices. 

Theorem 2 . 8 . Every communicating class C eC is of the form 

for some i eV. Vice versa, if C '■= 0^{i) n 0~{i) + 0 for some i eV, then C is a 
communicating class. 

Proof. Let C be a communicating class with i € C. Then since f ~ we have that 
C contains a path 7 = {i.. .i) and hence it follows that i € 0^{i) n 0~{i), that is, 
C c 0^{i) n 0~{i). Now consider j e 0^{i) n 0~(i) for some i e V. Then there 
exist a path 7 and n > 1 such that 7 ro( 7 ) = i and T^n(l) = j, as well as a path 7 ' 
such that 7 ro( 7 ') = j and TTmiY) = h for some m > 1. This immediately implies i ~ j 
for every element j 6 0^{i) n 0~{i), and therefore 0^{i) n c [i]. Conversely, 

assume that 0^{i) nO~{i) + 0 for some i^V. We have to show that 0^{i) nO~{i) 
is a communicating class, i.e. 0*{i) = [f]. Take j € nO~{i), then we 

argue as before that j ~ * and hence j € [j]. On the other hand, if j ^ (i) n 0~(i), 

then j 0'^{i) or j i In the first case there is no path from i to j, in the 

second case there is no path from j to i. Any of these two statements implies that 
i 'P j, which completes the proof. □ 

Definition 2 . 9 . A transitory vertex is a vertex that does not belong to a commu¬ 
nicating class. 

Note that by Theorem 12.81 transitory vertices are exactly those vertices i eV for 
which 0^(i) n 0~(i) = 0. This also means that 0^{i) n 0~(i) 4= 0 iff i e I 4 i-e., 
exactly these vertices anchor communicating classes. In addition, the statements in 
Theorem 12.81 take on this simple form because we have defined orbits in Definition 
O as starting with paths of length 1, not 0. If we include paths of length 0 in 
an orbit, then it always holds that i € 0^{i) n 0~{i). This trivial situation then 
needs to be excluded in Theorem 12.81 Similarly, we have defined communicating 
classes in Definition [23] using mutual access (a vertex i eV satisfies * € 14 if there 
exists a path of length > 1 from i to i). This avoids the triviality that each vertex 
communicates with itself. Note that for systems on continuous state spaces one 
needs separate non-triviality conditions, such as the existence of an infinite path 
within a communicating class and a condition on the richness of the orbits, see the 
discussion in [5], Chapter 3 for control systems. 

2.2. Communicating sets in L-graphs. We analyse communicating structure 
in graphs that admit limit behaviour. 

Definition 2 . 10 . In a directed graph G the out-degree of a vertex i eV is defined 
as the number of edges going out of the vertex i 

0{i) = #{(4j) : (b j) 6 E for some j e V}. 
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The in-degree of a vertex i corresponds to the number of edges coming into i 

I{i) = e E for some j € V}. 

Definition 2 . 11 . A graph is called an L-graph if every vertex has positive out- 
degree. 

From now on we only concentrate on a graph G = {V,E) such that 0{i) > 1 
for all i eV. L-graphs are needed to ensure the existence of various objects, such 
as communicating classes. Definition 12.111 states a non-degeneracy condition for 
orbits in a graph. We ensure the existence of communicating classes (with certain 
additional properties). 

Lemma 2.12. Each L-graph has paths of arbitrary length. 


Proof. Let G be an L-graph and n > 1. Pick ig e V, then by definition O(io) > 1. 
This ensures the existence of € F such that (foAi) is an incidence in G. Using 
the same argument we see that there is a vertex 12 e F and an incidence (ii,i 2 )- 
Continuing with this process up to step n we infer the existence of a path 

b — (^0; (ilU2); • ■ ■ Un-25 (fn—1 5 5 In) 


or equivalently, 


with ^( 7 ) = n. 


7 = (fo . ..in) 


□ 


Definition 2.13. A path 7 of length ^( 7 ) = n, with n > 1, is said to be a loop if 
there exists a vertex 167 such that 710 ( 7 ) = ^n( 7 ) = *■ 

Lemma 2.14. In a graph G with d vertices any path 7 of length £( 7 ) = n, with 
n> d contains a loop. 

Proof. We consider a path 7 in G such that 7 = {ioii ■ ■ - id) with £( 7 ) = d. Assume 
that the subpath {io ... id-i) contains no loop. Then all the vertices of {io ... id-i) 
are distinct and hence {io,... ,id-i} = V. Now id ^ V implies that there exists 
a 6 {0 ,... ,d - 1} with id = ia- Hence the subpath {ia. ■ ■ id) is a loop contained in 
7 . □ 


The next three lemmata explore the relationship between loops and communicating 
classes, leading to the existence of communicating classes in L-graphs. 

Lemma 2 . 15 . If G has a loop, then there exists a communicating class G in G 
such that the vertices in the loop are contained in C. 

Proof. Consider a loop A = (ioii. ■ ■ inio)■ Since the elements in A have mutual 
access each other, we have ik e [io] for 0 < k < n. Therefore each ik belongs to the 
same communicating class G = [io]. □ 

Lemma 2 . 16 . Let G he a graph and 7 = (io...in) a path in G. If there is a 
communicating class G with io e G in G, and if there is a € {1,... ,n} with ia i C, 
then ip for all /3 € {a,... ,n}. 

Proof. Using the notation of the statement of the lemma, assume, to the contrary, 
that there exists ft > a with ip e G. Then there are paths 71 = {io.. .ia) and 72 = 
{ia ■ ■ - ip .. .io), showing that io ~ ia and hence ia e G, which is a contradiction. □ 
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Lemma 2.17. If a vertex i belongs to a communicating class C in G, then there 
exists a loop X in C such that i € X. 

Proof. Let G be a graph and C a communicating class in G. Then there is pairwise 
communication between the elements in C, i.e., given i,j € G there exists a path 
= {i.. .j). In particular, for i = j we have the path ju = {i... i) with 7ro(7) = 
7 r„(j) = i for some n > 1 . Hence 7 is indeed a loop containing i. By Lemma | 2. 161 
all components of this loop are in G. □ 

Proposition 2.18. An L-graph has at least one communicating class. 

Proof. Consider an L-graph G with d vertices. By Lemma r 2. 121 there exists a path 
7 such that i(j) = n with n> d. By Lemma 12.141 the path 7 contains a loop A, and 
by Lemma 12.151 there exist a communicating class containing the vertices of A. □ 

Remark 2 . 19 . The proof of Proposition 12 . 181 actually shows the stronger statement: 
Let G be an L-graph and i e V. Then there exists at least one communicating 
class G with G c 0 ^{i). Note that, in general, i e 0 ^{i) may not hold. 

As a final idea of this section we explore an order on the set of communicating 
classes, which will lead to a characterisation of so-called forward invariant classes. 

Definition 2.20. Let G be a graph with a family C = {Gi,..., Gfc} of communi¬ 
cating classes. We define a relation on C by 

Gfj, <Gi, if there exists a path 7 e T" with 
710(7) e C't* and 7r„(7) € G„. 

Lemma 2.21. Let G be a graph with a family C = {Gi,..., G^} of communicating 
classes. The relation < defines a (partial) order onC. 

Proof. Reflexivity: Let G^ be a communicating class in G. By Lemma 12.171 for any 
i 6 Gfj. there exists a loop A in G^ such that i e X. Since G^ is a communicating 
class, the loop A gives us a path form G^ to itself, and therefore G^ ^G^. 

Antisymmetry. Assume that G^ :< G^ and G^, < C^. From the first relation we 
get a path 7^^ e T^’ for some p > 1 such that 

^ C'm and TTpi'Y^j.u) e G^. 

By the second relation there exists a path e F'^ for some q> 1 with 

TToili.^u) and 77,(7^^) 6 C^. 

Hence the concatenation {'yp,v,lvp,) is a path in G^ of length p + q. Since commu¬ 
nicating classes are maximal, and any two vertices in a communicating class have 
mutual access, it holds that = C^. 

Transitivity: Let C^, Q in C and suppose that G^ < Gi, and Gi, < hold. 
Since C^iG^, there exists a path 71 such that 7ro(7i) e G^ and 7r„j(7i) e G^ for 
some ni e N. Moreover, since Gv < Gj, there exists a path 73 such that 779(73) e Gv 
and 77^3(73) 6 G{ for some 713 e N. Since Gi, is a communicating class, there exists 
a path 72 in Gi, such that 719(72) = 7r„j^(7i) and T:n-i{l2) = 779(73) for some n2 e N. 
The path 7 = (71,72,73) is such that 779(7) ^ 77^(7) e G{ for m = 77.1-1-712+713. 

Therefore, G^ < G^. 

□ 
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Definition 2.22. A set of vertices ?7 of G is called forward invariant if, 

0^{U) c U. 


Similarly, U is called backward invariant if, 

o-{u) c U 


and invariant if. 


G+(t7)uG-(t7)ct/. 

Remark 2.23. Note that, by definition, a forward invariant communicating class is 
maximal with respect to the order < introduced in Definition 12.201 


Proposition 2.24. An L-graph G contains a forward invariant communicating 
class. 


Proof. Consider an L-graph G with set of communicating classes C = {Gi,..., Gfe}. 
Since an order relation on a finite set has maximal element, denote by G^ a maximal 
element in (C,<). We show that G^ is forward invariant: Assume to the contrary 
that Gfj, is not forward invariant. Since then G^ ) is not contained in G^,there 
exist 7, jo 6 V with i e G^ and jo i G^ such that {i,jo) e E. Since O(jo) > 1, 
there exists ji e V such that (jo,ji) e E. Note that by Lemma 12.161 b cannot 
belong to G^. By Remark [249] there exists a communicating class G c G^(jo) and 
G n G^ = 0 by Lemma [2. 161 ft follows that, G^ < G, which contradicts maximality 
of G^. □ 

Remark 2.25. Note that in the proof of Proposition I2.24| we actually showed the 
stronger statement: Let G be an L-graph and i^V . Then 0^{i) contains a forward 
invariant communicating class. 

Remark 2.26. Summarizing Remark |2.23l and Proposition 12.241 we see that for an 
L-graph the maximal elements of (C, <) are exactly the forward invariant communi¬ 
cating classes, ft also follows directly from Definition 12.201 that backward invariant 
communicating classes are minimal in (C,<). Note, however, that minimal commu¬ 
nicating classes in {€,<) need not be backward invariant. For this fact to hold we 
would need a backward nondegeneracy condition similar to the L-graph property, 
e.g., using the in-degree of vertices. For an analogue of this issue in the theory of 
control systems in discrete time compare [1]. 

2.3. Quotient graphs. There are at least two quotient structures associated with 
the idea of communicating sets in a graph. The first idea is to simply take the 
order graph given in Definition 12.201 Equivalently, this graph is obtained as the 
quotient The graph obtained from Definition 12. 201 does not necessarily cover 

all the vertices of a given graph G, and its edges may not be edges of G. For a given 
directed graph G = {V,E) we define Vq := C u E\t4 = {G, G is a communicating 
class} u {* 6 f/, i is a transitory vertex) as a set of vertices. The set of edges is 
constructed as follows: For A,B e Vq we set {A,B) 6 Eq if there exist i e A and 
j 6 i? with (i, j) 6 E. (Note the abuse of notation: if A e Vg is a (transitory) vertex 
of G then “i e A” is to be interpreted as “z = A”.) The graph Gg = (Vg,Lg) 
is called the extended quotient graph of G. ft is easily seen that the extended 
quotient graph of Gg is Gg itself. All vertices in Vg have specific interpretations 
in the context of Markov chains, see Section [57^ 
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3. The semiflow of a finite directed graph 


Our second approach to study decompositions of graphs is based on an idea 
from the theory of dynamical systems. A Morse decomposition describes the global 
behaviour of a dynamical system, i.e. the limit sets of a system and the flow between 
these sets. The following definition and basic ingredients are standard. These can 
be found for example in mi- 

Definition 3 . 1 . A Morse decomposition of a flow on a compact metric space X 
is a finite collection {A4i, i= l,...,n} of nonvoid, pairwise disjoint, and compact 
isolated invariant sets such that: 

n 

• For all a:: 6 X one has a;(a:), uj*{x) c 

2=1 

n 

• Suppose there are and xi, ...,x; € Av^jAfi with c 

i=l 

and w(xi) c Atfor i = 1 ,..., 1 ; then Mjg + Mp. 

The elements of a Morse decomposition are called Morse sets. 

A Morse decomposition results in an order among the components, the Morse 
sets, of the decomposition. It can be constructed from attractors and repellers, 
and the behaviour of the system on the Morse sets is characterised by (chain) 
recurrence. Here we then construct an analogue for (discrete) systems defined by 
L-graphs. Unfortunately, this analogy is not complete since systems defined by 
these directed graphs only lead to semiflows, i.e. systems for which the time set is 
N, and not all of Z. 

3.1. Semiflows associated with graphs. When adapting the idea of Morse de¬ 
compositions and attractors-repellers to systems induced by directed graphs, one 
faces two main challenges: The first concerns the topology on discrete spaces that 
has some interesting consequences for limit sets, isolated invariant sets, etc. The 
second challenge stems from the fact that the out-degree of vertices can be > 1, 
resulting in set-valued systems. This is the reason why graphs define semiflows (on 
N instead of Z). 

Let G = {V,E) be a finite directed graph. We denote by V{V) the power set of 
the vertex set. We consider the topology derived by the discrete metric on U(U). 

The directed graph G gives rise to two semiflows, one with the positive integers 
N as time set, and one with the negative integers N“ := {-n, n e N}: 

:NxU(U) ^U(U), 


(3.1) $G(n,A) 

Similarly, we define 


j e U : 3 i e A and 7 e F” such that 

710 ( 7 ) = * and 7 r„( 7 ) = j 

X V{V)^V{V), 


(3.2) 


$e(n,A) 


j € U: 


3 


* € A and 7 € F " such that 
7 ^ 0 ( 7 ) = i and 7r_„(7) = i 


We note that the definition of the semiflows ^cin, A) and ^^{n, A) only requires 
the basic ingredients of a graph: the sets of vertices and of edges. 

The next proposition collects some properties of the maps defined in (13.111 and 

( 1321 ). 
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Proposition 3.2. Consider a graph G and the associated map defined in (13.ip . 
This map is a semiflow, i.e. it has the properties: 

• i&G is continuous, 

• $G(0,^) = ^for aWAeViV), 

• $G(n + m,A) = for all ^ rn,neN. 

The same properties hold for the negative semiflow 

Proof. Note that in the discrete topology every function is continuous. The second 
item holds by definition of T*^ in m-- The vertices reached from A under the flow 
at time zero, are the elements in V that belong to A. To show the third item we 
first assume that the three sets $g(w. A), ^a{n + m,A), and $g(r-, are 

nonempty. Consider j € $G(n + m,A): There exist i e A and a path a 6 such 
that 7 ro(a) = i and 7r„+m(Q!) = j- We split a as the concatenation of two paths (3 and 
7 with £(/3) = m and £( 7 ) = n, having, tto^P) = i, TTmiP) = k and 719 ( 7 ) = k, 7 r„( 7 ) = j 
for some k eV. Observe that by definition we have k e $g(to,A) and hence j e 
$G(n, $G(rR,^))- On the other hand, if j € $G(n, $G(m, A)) then there exist k € 
^a{m,A) and /3 € T" such that 7 ro(/ 3 ) = k and 7 r„( 7 ) = j € ‘hG(™,^))- Since 

k 6 ^Q{m,A), there are A and a € T™ such that 719 ( 0 !) = i and iTmio) = k. The 
concatenation 7 = (o, /3) satisfies 719 ( 7 ) = * S'^.d TTm+n{l) = j, hence j € $G(n+m, A). 

To finish the proof for $g, we consider the case that (at least) one of the three 
sets $g(w,A), $g( 7I)‘hG(™, A)), and + m,A) is empty: Note first of all 

that by the definition of in (| 3 . 1 I 1 we have $G(n, 0 ) = 0 for all n € N. (i) If 
$g(to,A) = 0 then we have $G(n, $G(m, A)) = 0 by the preceding argument. 
Now if $G(n + m,A) + 0, then we can construct, as in the previous paragraph, a 
point k € $G(m,^) by splitting a path a e with 719(0) e A and 7r„+m(o) e 

<l>a(n + m, A). This contradicts ^g(^,A) = 0 and therefore + m,A) = 0 . 

(ii) Assume next that ^G(n,^G('^,A)) = 0 . Using the same reasoning from the 
previous paragraph, we see that if j € $g( 7 i + to. A) then j e $G(n,$ g(to. A)). 
Hence it holds that $G(n- + to. A) = 0. (in) If $G(n + to, A) = 0 then there exists 
no path 7 € with 719(7) ^ But by the reasoning in the previous paragraph, 

if j 6 $G(n, $g(to. A)) then there exist * 6 A and 7 € such that 719(7) = * und 
T^m+nil) = j, which cannot be true, and hence we see that $G(n, $g(to. A)) = 0. 

The proof for follows the same fines. □ 

Remark 3.3. If G is an L-graph, then $g(u,A) + 0 for A ^ 0 and n € N. This 
observation may not hold for the negative semiflow without additional assump¬ 
tions. 

One might wonder if the semiflows $g defined in (13.11) and <I>g from (13.21) can 
be combined to a flow on Z. It is not hard to see that in general this is not possible 
even if the graph G has additional properties (such as being an L-graph) or if one 
restricts oneself to graphs that are one communicating class. 

Remark 3.4. The semiflow as defined in (13.2p can be interpreted as the positive 
semiflow of the graph = (U, A^), where (f, j) e iff (j, i) e E. is sometimes 
called the time-reverse semiflow of <I>g. Under the corresponding assumptions, all 
statements for a positive semiflow also hold for its time-reverse counterpart. 

3.2. Morse decompositions of semifiows. We next turn to some concepts from 
the theory of dynamical systems and study their analogues for the semiflows defined 
in dSI]) and (IT^ . Note the state space 7^(U) of the semiflow $g is finite with the 


10 


JOSE AYALA AND WOLFGANG KLIEMANN 


discrete topology. It suffices to introduce the concepts for points A eV{V). To 
avoid trivial situations where A) = 0 for some n € N and A € 'P{V), A ^ 0, we 

assume that all graphs are L-graphs, compare Remarks 13.31 above and 13.81 below. 

Next we adapt the necessary ingredients for a meaningful Morse decomposition 
of a semiflow in an L-graph. 

Invariance: A point A e 'P{V) is said to be (forward) invariant if ^ain, A) c A 
for all n 6 N. Note that A e 'P{V) is invariant under $g iff A is a forward invariant 
set of the underlying graph G = {V,E), compare Definition 12.221 Hence invariance 
under $g is a fairly strong requirement of a set A € P(H). As we will see, a 
meaningful Morse decomposition of the semiflow $g only requires a weak form of 
invariance. 

Definition 3.5. A point A € V(y) is said to be weakly invariant if for all n € N 
we have A) n A 0. 

Isolated invariance: For a (forward) invariant set A € V{V) one could define 
forward isolated invariant. But because of the discrete topology, we could choose 
N{A) = A, and any forward invariant set then satisfies this property. As we will 
see, because of the discrete topology employed, a meaningful Morse decomposition 
of the semiflow <I>g does not require the property of isolated invariance. 

Limit sets: To adapt the concept of a limit set from the standard definition to 
the semiflow note that a sequence converges in the discrete topology iff it is 
eventually constant. Hence limit sets can be defined in the following way: 

Definition 3.6. The oj-limit set of a point A e 'P{V) under $g is defined as 



Remark 3.7. Note that by definition of the discrete topology we have for A e 'P{V) 
the fact a;(A) = u{w({i}), i e A}. 

Remark 3.8. The existence of w-limit sets and the L-graph property are closely 
related: Let G be a graph and its associated semiflow. Then G is an L-graph 
iff w(A) + 0 for all A e 'P{V). This observation justifies our concentration on 
L-graphs in this section. 

For continuous dynamical systems Morse decompositions are required to contain 
all of the w, a;*-limit sets of the system. For semiflows induced by graphs a weaker 
condition of recurrence turns out to be appropriate: 

Definition 3.9. Let G be an L-graph with associated semiflow $g on V{V). A 
one-point set {i} e V{V) is called recurrent, if there exists a sequence n; in N, 
m oo, such that {f} c $G(ni,{i}). A set i? e V{V) is called recurrent if for 
each i € B the one-point set {*} is recurrent under $g. The set TZ := {i ^ V, {i} 
is recurrent} is called the recurrent set of <f>G- H TZ = V the semiflow $g is called 
recurrent. 

Note that by Definition 13.91 it holds that {*} € V{V) is recurrent iff * € w({f}) iff 
i € A for some loop A of G. 

No-cycle condition: For continuous dynamical systems the no-cycle property 
(second item in Definition 13.111 is essential for the characterization of a Morse de¬ 
composition via an order. For semiflows induced by graphs we can formulate an 
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analogue of the no-cycle condition either using only the (forward) semiflow or 
a combination of and 

Definition 3.10. Consider the semiflow and a finite collection A = {^i, An} 
of points inV{V). .A is said to satisfy the no-cycle condition for i1>g if for any 
subcollection of A with oj{Aj^) n 0 for a = 0,...,/-! it holds 

that Ajg + Aj^. 

Remark 3.11. Alternatively, we can define A € V{V) to be a no-return set if for 
all one-point sets {*} € V{V) we have: If a;*({j}) n A 0 and w({i}) n A 0 then 
{i} c A, where uj*{B) is the w-limit set for B € 'P{V) under the negative semiflow 

With these preparations we can now introduce our concept of a Morse decom¬ 
position of the semiflow 


Definition 3.12. Let G = {V,E) be an L-graph. A Morse decomposition of the 
semiflow oiiV(V) is a finite collection of nonempty, pairwise disjoint and weakly 
invariant sets {Mf^ e V{V) ■ fJ.= 1 ,..., k} such that: 

• {Aifj, 6 'P{V) : /I = 1,..., k} satisfies the no-cycle condition from Definition 

ISAOl 

The elements of a Morse decomposition are called Morse sets. 


Proposition 3.13. Let G = {V,E) be an L-graph and let 

be a finite collection of nonempty, pairwise disjoint and weakly invariant sets of the 
semiflow $<3 on V{V}. The collection Ad is a Morse decomposition of iff the 
following properties hold: 

• 7^c 

• The relation defined by 

U < Alo =MpmM 

“ ^ with a;(AdjJ n 0 for f = 0, ...,Z - 1 

is a (partial) order on M . 

We use the indices /i = 1,..., fc in such a way that they reflect this order, i.e. if 
Ma i Mp then a< p. 


Proof. Assume first that M = {Ad^x e V{V) : /r = 1,..., k} is a Morse decomposition, 
we need to show that the relation is an order. Reflexivity: Let Ad^ e Ad, then 
Ad^ is weakly invariant, i.e. ^a{n,A) n A ^ 0 for all n € N. Since Ad^ consists of 
finitely many elements there is at least one i € Ad^ such that i 6 $G(nfc, Ad^) nAd^ + 
0 for infinitely many Uk e N. Hence i € a;(Ad^)nAd^ and therefore w(Ad^)nAd^ + 0, 
which shows Ad^ < Ad^. Antisymmetry: Assume there are Ad^, Ad /3 € Ad with 
Ada < Ad /3 and Mp < Ma- This means there are = Ad^, Adj^, ...Adj, = Mp in 
Ad with uj(MjJ n Adj 3 ^^ 0 for f = 0, ...,l - 1 and Mko = Mp,Mki, = Ma 

in Ad with w(AdjJ n Adj 3 ^^ =#= 0 for i = 0, ...,to - 1. If there were two different sets 
in the collection Adjj, = Ada, Ad...Ad = Mp, Mko = Mp,Mki, ■■■Mkm = Ma, 
then Ma + Ma, which cannot hold. Hence all sets in this collection are the same, 
in particular Ada = Mp. Transitivity: This follows directly from the definition of 
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Assume now that M. = {M/j. e 'P(^) '■ fJ-= 1,fc} is a finite collection of nonempty, 
pairwise disjoint and weakly invariant sets such the relation is an order. We 
need to show that A4 satisfies the no-cycle condition: If A4jg, is a subcol¬ 
lection of A with uj(Aj^) n 0 for a = - 1 then A4jg < A4j,. If this 

subcollection is disjoint, then JAjg $ Mji , in particular Mjg + . □ 

As in the case of continuous dynamical systems, Morse decompositions for semi¬ 
flows induced by L-graphs need not be unique. For instance, the collection {F,0} 
always is a Morse decomposition of any $< 3 . As is the case of continuous dynamical 
systems we can use intersections of Morse decompositions to refine existing ones, 
compare page 8 in [4]. Since the sets V and 7^(F) are finite, the semiflow on 
T’(V) admits a (unique) finest Morse decomposition for any L-graph G. The next 
result characterises the finest Morse decomposition. 

Theorem 3.14. Let G = {V,E) be an L-graph with associated semiflow <I>g on 
V(V). For a finite collection of nonempty, pairwise disjoint sets 

the following statements are equivalent: 

• Ai is the finest Morse decomposition of $< 3 . 

• M.=C, the set of communicating classes of G, compare Lemma \2.21[ 

Proof. Assume first that M = e V{V) ■ fJ.= 1 ,..., k} is the finest Morse decom¬ 
position of and let x,y e Mfj, for some fj, = l,...,fc. We have to show that x 
and y communicate. Observe first that x e implies that there exists a loop 7 
of the graph G with x e 7 . If there is no such loop then {AI^\{x},A1q ■ a + fj,} 
is still a Morse decomposition and hence Ai cannot be the finest one. If x does 
not communicate with y, take the communicating classes [x] 0 and [y] + 0 , 

[x] n [ 2 /] = 0 together with set T = {A : A is a loop not in [x] u [j/]}, to form the new 
Morse decomposition Ai' = {[x\,[y^, a '■ 0 . + y}, which is finer than the given 
Ai, leading to a contradiction. 

To see the converse, let C = {Gi,...,Ck} be the set of communicating classes of 
the graph G. The Ga are clearly nonempty, pairwise disjoint and weakly invariant 
for all a = 1,..., k. Recall that {i} e V{V) is recurrent iff i e A for some loop A in G, 
and therefore we have 7 ?. c Finally, Lemma [ 2.211 shows that the relation 

“<” defined in Proposition 13.131 is indeed an order relation. Hence, the two ordered 
sets (C,<) and {A4,<) agree. □ 

Remark 3.15. The proofs of Proposition 13.131 and of Theorem 13.141 show the re¬ 
lationship between w-limit sets of <1 >g and loops of G: For each A € V(V) the 
limit set uj{A) contains at least one loop. And vice versa, if * € A is a vertex of a 
loop A of G, then i e a;({i}). This shows that for the finest Morse decomposition 
A4 = {Ai^, Ai = 1, k} of 4>(3 we have: 

= {* € A,A is a loop of G} c u{uj{A),Ai V{V)}. 

This situation is different from the one for continuous dynamical systems, where 
Morse sets may contain points that are not contained in limit sets, see [4], Example 
5.11. Indeed, it is not hard to see that for discrete semifiows not all points in 
w-limit sets need to be elements of a Morse set. 
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Remark 3.16. Consider an L-graph G with associated semiflow <i>G- It follows from 
Theorem 13.141 that i € u{a;(A), A 6 V{V)} \ M.p, is a finest Morse set} iff 

f is a transitory vertex and there exists a (finest) Morse set M. with % e $< 3 ( 71 , At) 
for some n > 1 . 


3.3. Attractors and recurrence in semiflows. Next we adapt the concept of 
an attractor to the semiflow on an L-graph and analyse the connection with Morse 
decompositions. 

Definition 3.17. Let G = {V,E) be an L-graph with associated semiflow <1 >g on 
P(C). A point A 6 'P{V) is called an attractor if there exists a set N c V with 
Ac N such that = A. 

A set N as in Definition 13.171 is called an attractor neighborhood. Note that in 
the discrete topology A is a neighborhood of itself and hence a point A 6 V{V) is 
an attractor iff U!{A) = A. We also allow the empty set as an attractor. 

A definition of repellers for semiflows of graphs is not obvious, but the idea 
of complementary repellers from [3] carries over with an obvious modification for 
semiflows: 


Definition 3.18. For an attractor A 6 'P{V), the set 

A* = {ie V, a;({ 7 })\A + 0 } e r(V) 

is called the complementary repeller of A, and (A, A*) is called an attractor-repeller 
pair. 


Morse decompositions of semiflows can be characterized by attractor-repellers 
pairs, in analogy to [J]. 


Theorem 3.19. Let G = {V,E) be an L-graph with associated semiflow $<3 on 
V{V) . A finite collection of sets M. = eV{V) ’■ fi= 1,..., fc} defines a Morse 
decomposition of^c if and only if there is a strictly increasing sequence of attractors 

0 = Aq c Ai c A 2 c ... c A„ c V, 


such that 

Mn-i = Ai+i n A* for 0 < i < n - 1. 


Proof. Recall the indexing convention for Morse sets from Proposition 13.131 

Let M = {M/j, € V{V) ■ fJ,= 1, ..., k} be a Morse decomposition of In analogy 
to the continuous time case we define the sets A^ for k = 1 , ...,n as follows: 

Afc = {x € P, cc (x) n (Ain LI... u Ain— fc+i) ^ 0}■ 

Note that for semiflows of graphs we have Ak = C>^(AlnU...uAln-fe+i). We first need 
to show that each Ak is an attractor. The inclusion uj{Ak) c Ak follows directly 
from the characterization above of A^ as a positive orbit. To see that Ak c u!{Ak) 
pick X e Ak. Then there exists /xe{n-fc + l,...,n} with x e C>^(Al;j). According to 
Theorem [Xm the set Al^^ is a communicating class of the graph G and hence every 
element 2 : e Al^ is in a loop 7 that is completely contained in Al^, compare Lemma 
12.171 Therefore there is 2 : e A^ with x e for as sequence ni 00 , i.e. 

Ak c uj{Ak). Hence each Ak is an attractor. 

Next we show that Adn-i = A^+i n A* for 0 < i < n - 1. To see that Ain-i ^ A^+i 
pick X € A4 n-i- Since A4 n-i is a communicating class of the graph G we have 
a;*(a:) n Aln-i + 0 and therefore x € A^+i. To see that Ain-i ^ A* assume that there 
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exists X € Mn-i with x ^ A*, i.e. uj{x)\Ai = 0 or uj{x) c Ai. But x € Mn-i means 
uj(x)nAin-i + 0 , and by definition we have M.n-i<^Ai = 0 , which is a contradiction. 
This shows M.n-i ^ Ai+i n A* for 0 < i < n - 1. To see the reverse inclusion, let 
X e Ai+i nA*, i.e. x e and u:{x)\Ai + 0 . Recall that by Remark 

I3.15l a;fa;l contains a loop 7 of the graph G, and by Lemma [2.171 and Theorem l3.14l 
each loop is contained in a Morse set. Hence ijj{x) n (A4„ u ... u Mn-i) + 0, which 
by definition of a Morse decomposition means that x e Mn-i- 

Let Mn-i = Ai+i n A* for 0 <z<ti- 1 be defined as in the statement of the 
theorem. We have to show that {Mi,...,Mn} form a Morse decomposition. We 
start by proving that the sets Mn-i = ^i+i <^A* are nonempty. Note first of all that 
Ai, ■..,An 0 by assumption. We have by definition of attractor-repeller pairs that 

V = Aq ^ Al ^ ^ A^_i D H*. Now A^_i + 0 can be seen like this: If = 0 

then for all x € R we have w(x)\H„_i = 0 , i.e. u}{x) c An-i. Hence there is m € N 
such that for all a > m we have $< 3 ( 0 !, V\An-i) c and therefore cannot be 
an attractor. We conclude that Aq, A^,..., A^_^ + 0 . Now if Ai+i A* = 0 then we 
have by the same reasoning as before: For all x e Ai+i it holds that uj{x)\Ai = 0 , 
i.e. uj(x) c Ai and Ai+i cannot be an attractor. 

The sets Mi are pairwise disjoint: Let a < /3, then M„_q, n = Aa+i nAln 
Ap+i nA*p = Aa+i n A*p c Af} n A*p = 0. 

The sets Mi are weakly invariant: As above, it suffices to prove that Mn-i = 
Ai+i nA* contains a loop of the graph G. If there is no loop in n A*, then there 
exists TO 6 N such that for all a > to we have $ 0 ( 0 :, Ai+i n A*) c A^ and therefore 
Ai+i cannot be an attractor. 

The collection {Mi,...,Mn} satisfies the no-cycle condition: This is just a re¬ 
statement of the assumption that Aq c Ai c A 2 c ... c A„ is a strictly increas¬ 
ing sequence of attractors. We have shown so far that M = {Mi,...,Mn} sat¬ 
isfies the conditions of a Morse decomposition, except for 77. c Now let 

M' = {M'l, ...,M'i.} be the finest Morse decomposition of $( 3 . Since the recurrence 
condition was not used in the first part of the proof of Theorem 13.141 and since 
{i} eV{V) is recurrent iff i e A for some loop A of G, we know by Lemma [2.151 that 
TZcu^^^M'^ c Altogether we see that M = {Mi,...,Mn} is a Morse 

decomposition. □ 

Corollary 3.20. Let M = {M/^ e V{V) : /r = 1, ...,fc} be the finest Morse decom¬ 
position of a semiflow 4 )<3 on 7^(R), with order <. Then the maximal (with respect 
to <) Morse sets are attractors. Furthermore, the smallest (with respect to set 
inclusion) non-empty attractors are exactly the maximal (with respect to <) Morse 
sets. 

Proof. If M is a maximal Morse set of the semifiow $( 3 , then, according to Theorem 
13.141 and Proposition 13.131 AI is a maximal communicating class of the graph G. 
Hence AI is forward invariant, uj{M) = AI and AI does not contain any attractor, 
except for the empty set. 

Vice versa, if A is a smallest (with respect to set inclusion) non-empty attractor, 
then A is a Morse set according to Theorem l3.191 If A is not maximal (with respect 
to <), then A is not forward invariant for the graph G and hence there exists a point 
X e G^(A)\A such that 0*(x) n A = 0 (by Lemma [2.1611 . According to Remark 
12.251 0*{x) contains a maximal communicating class, which is an attractor A' ^ A 
and hence A is not a smallest non-empty attractor. □ 
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It remains to analyse the behaviour of the semiflow on a Morse set. Definition 
13.91 and Remark 13.151 already point at a recurrence property that holds for w-limit 
sets: Note that by Definition 13.91 it holds: {i} e 'P{V) is recurrent iff f e a;({i}) iff 
f € A for some loop A of G. Hence we obtain from Remark l3.15l for the finest Morse 
decomposition M = {Ad/x, A* = 1 ,..., k} of 

(3.3) TZ = ul^,M^. 

The recurrent set is partitioned into the disjoint sets of the finest Morse decompo¬ 
sition under the following natural concept of connectedness. 

Definition 3.21. A set B € V{V) is called connected under <I>g if for any i,j^B 
there exist n 6 N and a map p ■ {0, ...,n} ^ B with the properties 

• P{0) = i, p{n) = j 

• p{m + 1 ) 6 $ 0 ( 1 , {p(w)}) for m = 0, ...,n - 1 . 

The flow is called strongly connected if the set of vertices V is connected 
under $g. 

The following result then characterises the behaviour of the semiflow on its 
Morse sets, compare Theorem 6.4 in [4] for continuous dynamical systems. 

Theorem 3.22. Let G = {V,E) be an L-graph with associated semiflow $(5 on 
V{V). The recurrent setTZ of satisfies 

7^ = ^{AuA^ A is an attractor} 

and the (finest) Morse sets of <I>g coincide with the ^c-connected components of 

n. 

Proof. Assume that x e TZ, then a: e 7 for some loop 7 of the graph G. Let A be 
an attractor for then if a: e A we are done. Otherwise if a: ^ A then it holds 
that 7 n A = 0 . But 7 c a;(a:) and therefore a;(a:)\A + 0 , which means that x e A*. 
Conversely, if a: e n{AuA*, A is an attractor}, then x is in any attractor containing 
a;(a:). Arguing as in the proof of Theorem 13.191 there exists a loop 7 of the graph 
such that a: 6 7 , which shows that x €TZ. 

The second statement of the theorem follows directly from Definition 13.211 and 

m- □ 

As discussed in the paragraph about invariance (see Section [3^, forward invari¬ 
ance under is a fairly strong requirement for a set A c V, and thus it appears 
that there are few sets to which one can restrict the semiflow <I>g, namely (unions 
of positive) orbits. 

Definition 3.23. Let G = {V,E) be an L-graph with associated semiflow on 
V{V). Let G' = (V',E') be the subgraph of G for a subset of vertices V' c V. The 
resulting semiflow ^c’ on V{V') is called the semiflow restricted to V. 

Note that if Ad c 1/ is a Morse set of 4 >g, then the induced graph (Vm.Em) is 
an L-graph. This observation allows us to prove the following fact about Morse 
sets and recurrence. 

Corollary 3.24. Under the conditions of Theorem l3.22l the semiflow <I>g restricted 
to any Morse set is recurrent. 


Proof. The proof follows directly from Theorem l3.14l and Definition 13.91 


□ 
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As we have seen, most of the concepts used to characterise the global behaviour 
of continuous dynamical systems can be adapted in a natural way to the positive 
semiflow of an L-graph, resulting in very similar characterisations. Indeed, the 
proofs for semiflows on a finite set are considerably simpler than the corresponding 
ones for continuous dynamical systems. What is missing in the context of semiflows 
is first of all the group property of a flow, and hence limit objects for t ->• -oo. This 
results in missing some of the invariance properties of crucial sets, such as limit sets, 
Morse sets, the (components of) the recurrent set, etc. And secondly, the use of the 
discrete topology implies that while all points in the (finest) Morse sets are limit 
points, not all w-limit points of the semiflow are contained in the (finest) Morse 
sets. But those exceptional limit points (and hence the set of all limit points) can 
be characterised, compare Remark [3. 161 

4. Matrices associated with graphs and their semiflows 

In the previous sections we have analyzed the communication structure of graphs 
using two different mathematical languages, that of graph theory and that of dy¬ 
namical systems. In this section we will briefly use yet another language, matrices 
and linear algebra. Connections between graphs and nonnegative matrices have 
been studies extensively in the literature, compare lamiin]. We will describe some 
connections and hint at algorithms that allow for the computation of the objects 
discussed in the previous sections. 

Definition 4.1. Given a graph G = {V,E) with jfV = d. The adjacency matrix 
of G is the dx d matrix with elements 



1 if(*,i)€A 


0 otherwise. 


Vice versa, denote the set of dxd matrices whose entries are in {0,1} hy M{d, {0,1}). 
Any matrix A e M((i, {0,1}) is called an adjacency matrix and can be viewed as 
representing a graph. 

Definition 4.2. An adjacency matrix is called an L-matrix if each row has at 
least one entry equal to 1 . 

Remark 4.3. Alternatively, the entries Uij € Ac can be viewed as paths in G of 
length 1 : aij = 1 iff there exists an edge 7 e T^ with 719 ( 7 ) = * 711 ( 7 ) = j. 

Continuing this thought we have the following relationship between paths of length 
n > 1 and entries of A'q, the n-th power of Aq'- € A'q is exactly the number 
of (different) paths 7 e T" from i to j in G. Hence the f-th row of Aq describes 
exactly the vertices that can be reached from i via a path of length n. In complete 
analogy, the i-th row of (A^)" describes exactly the vertices from which i can be 
reached using a path of length n. 

The communication concepts developed in Section lOl onlv depend on the existence 
of paths connecting certain vertices and not on the number of such paths. We use 
the standard Boolean addition +* and multiplication to describe these ideas. We 
extend the Boolean addition and multiplication for matrices in M((i, {0,1}) in the 
obvious way, denoting by A”* the n-th Boolean product of A e M(d, {0,1}) with 
itself. Note that M(d,{0, 1}) is closed under Boolean addition and multiplication. 
Since computer calculations involving +* and are very fast, we obtain efficient 
algorithms for the computation of orbits and communicating classes. 
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Let G = {V,E) he a, graph with = d and adjacency matrix A. For a vertex i eV 
its positive and negative orbits are given by 

= {j ^ = 1 for some n= 1, ...,d} 

0~(i) = {j e V, = 1 for some n= 

The proof of these facts follows directly from Section 12.11 and Remark 14.31 above. 
The communicating classes of G can now be computed as C = 0^{i) n 0~{i) for 
i e V, compare Theorem 12.81 The order among communicating classes can be 
determined directly from the computation of 0^{i) (or compare Remark 

12. 191 and Definition 12.201 Communicating classes and their order are also sufficient 
to compute the quotient graphs Gq and Gq of a given graph G. Hence all the 
objects analyzed in Section [2.II can be computed effectively using the matrix ideas 
described above. 

The concepts of irreducibility and aperiodicity play an important role in the analysis 
of nonnegative matrices. We briefly introduce these concepts and discuss their use 
in the analysis of communication structures in graphs. 

Definition 4.4. A matrix A e M{d, {0,1}) is said to be irreducible if it is not 
permutation similar to a matrix having block-partition form 

I All Ai2'\ 

\ 0 A22 j 

with All and A 22 square. 


Lemma 4.5. Let G be a graph with adjacency matrix A. Then A is irreducible ijj 
G consists of exactly one communicating class. 


A proof of this lemma can be found, e.g., in [7]. Note that the adjacency matrix of 
any L-graph is permutation equivalent to a matrix of the form 


(4.1) 


'’All Ai2 
0 A22 


Aq = 


Aii' 

A 2 i 


\ 0 ■ ■■ 0 All) 


where the square blocks An, i= 1,...,Z correspond to the vertices within the com¬ 
municating class Gi for i = l,...,k and to the transitory vertices, and the blocks Aij 
for j > i determine the order structure among the communicating classes. Hence 
Aq “is” the adjacency matrix of the extended quotient graph Gq, compare Section 
12.31 For additional characterizations of irreducible nonnegative matrices we refer 
to [15], Chapter 9.2, Fact 2. 


Definition 4.6. Let Ae M{d, {0,1}) be irreducible with associated graph G. The 
period of A is defined to be the greatest common divisor of the length of loops of 
G. If this period is 1, the matrix is said to be aperiodic. We say that a graph is 
periodic of period p (or aperiodic) if its adjacency matrix has this property. 


Lemma 4.7. A matrix A e M((i, {0,1}) is aperiodic iff > 0 (has all elements 
> 0} for some n e N. 
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A proof of this lemma can be found, e.g., in [7]. For additional characterizations 
of aperiodic matrices we refer to [15], Chapter 9.2, Fact 3. One can extend this 
definition to any communicating class of a graph G: Let G c F be a communicating 
class of G and Ac its diagonal block in the representation m of the adjacency 
matrix Ac- Note that Ac is irreducible and we define the period of G to be the 
period of Ac- 

The rest of this section is devoted to studying some of the connections between the 
semiflow of a graph and the adjacency matrix. Since the semiflow is a sequence of 
maps •) • 'Piy) n € N, we first need to define the analogue of ViV). 

For a graph G = {V,E) with #(G) = d, we can proceed as follows: 

For a subset A c F let xa denote its characteristic function, i.e. 


XA(i) = 


1 if i € A 
0 ifi^A. 


Let Ci be the i -th canonical basis vector of Define t: ViV) by 

''(^) = Tfi=iXA{i)ei. 

We denote by the (vertex set of the) unit cube in Note that t: V{V) 
is bijective and hence we can identify 'P{V) with as sets. We will use the same 
notation for the two versions of the map l. 

With these notations we can express the semiflow fl>c in terms of the adjacency 
matrix A: For paths of G of length 1, i.e. for edges we have Oi(i) := {j € F, (i,j) e 
Ej = A). The set Oi(i) is also called the orbit of i at time 1. Similarly 

we have for IF c F: (IF) := {j e V, (i,j) e E for some i 6 IF} = t“^(r(lF)^ •* A). 

By Remark 14.31 we obtain for paths of length n > 1 

" ( that 7ro(7) = i and 7r„(7) = J J 

The definition of the associated (positive) semiflow in Equation (13.11) now yields 
the following alternative way of describing this semiflow 

( 4 . 2 ) ^G{n,W) = ■* A^*f). 


This observation justifies the following definition: 

Definition 4.8. Consider a matrix A e M(d, {0,1}) and let c R'^ be the 
d-dimensional unit cube. The map 

is called the positive semiflow of A. Similarly, the map 

4-^ : N- X ^ 

^-A{n,q) = {q^-* {A^rf 

is called the negative semiflow of A. 

It follows from (14.21) . Proposition l3.2l and from bijectivity of i ■ V{V) that 'I'a 
and are, indeed, semiflows. This allows us to reinterpret all concepts and results 
from Sections 13.11 - [37^ for semiflows of square {0, l}-matrices. Alternatively, we 
could have developed the theory for semiflows of the type and then translated 
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the results to graphs. The key condition in Sections 13.1113.31 is for a graph to be an 
L-graph, which translates into L-matrices, see Definition 14.21 
To complete this chapter, we mention a few connections between the semiflow 
of an L-graph and concepts from matrix theory: 

Let G = (V, E) be an L-graph with associated semiflow $ : N x V{V) V{V). Let 
A be the adjacency matrix of G with associated semiflow : N x Then 

it holds: 

(1) G has exactly one communicating class iff <1> has only the trivial Morse 
decomposition { 0 ,D} iff A is irreducible. 

(2) A vertex j 6 D is in a communicating class C c V iS {*} is recurrent under 

$ iff > 0 . 

(3) For two communicating classes < G^, holds iff their corresponding Morse 

sets satisfy iff the adjacency matrix A^^ of the subgraph corre¬ 

sponding to G^ has a smaller index in the representation (14.ip than A^^. 

In the next chapter we will study Markov chains and interpret many of the results 
we have obtained so far in that context. We prefer to include some elementary 
results and definitions on Markov chains to facilitate exposition. 

5. Characterization of Markov Chains via Graphs and Semiflows 

Markov chains are discrete time stochastic processes for which the future is 
conditionally independent of the past, given the presence. If the state space of a 
Markov chain is a finite set, its probabilistic behavior can be analyzed using specific 
graphs and/or matrices. The goal of this section is to utilize the concepts and theory 
developed in Section 3 for the analysis of finite state Markov chains. This allows us 
to restate some well-known properties of Markov chains using graphs and semiflows, 
and to show a few new connections. We end up this work by setting up a “three- 
column dictionary”of equivalent objects and results in the three “languages”. 

5.1. Review of Markov chains. In this section we present without proofs some 
standard results in the theory of Markov chains. We refer the reader to [isiiiiiiz] 
for more details on finite chains, and to [5] Chapter 14.1, for a thorough discussion 
of chains on countable state spaces. 

Let (D,.F,P) be a probability space and S a finite set with cardinality #{S) = d. A 
discrete time stochastic process on D with values in the state space S' is a sequence 
of random variables : D S, n € N. For each w € D the sequences (A„(w), 
n e N) are called trajectories of the process. Recall that a Markov chain with 
values in S is a discrete time stochastic process satisfying the Markov property, i.e. 

— j I Aq — Zq, ■ • ■: A^n -1 ~ ^n- 1 : A^n ~ ~ — j \ Aji — z} 

for all times n e N and all states zq, s S. 

Recall the so-called Chapman-Kolmogorov equation for the n-step transition 
probabilities is given by: 

(5.1) p„(z,j) = P{A„ = j I Ao = z} = Y,Prihk)pn-r{k,j) 

ksS 

for 1 < r < rz. 

Definition 5.1. A state z € S has access to a state j e S' if Pn{i,j) > 0 some 
n> 0. A state z e S communicates with a state j 6 S if Pn{i,j) > 0 and Pm{j,i) > 0 
for some n,m> 0. 
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Definition 5.2. For a state ie S we define its period by 

S{i) = gcd{n > 1, > 0}, 

where gcd denotes the greatest common divisor. Then i is called periodic if 6{i) > 1, 
and aperiodic if 6{i) = 1. A Markov chain is said to be aperiodic if all points i e S 
are aperiodic. 

A crucial idea in the analysis of the qualitative behavior of stochastic processes is 
that of reachability, i.e., trajectories starting at one point reach (a neighborhood 
of) another point. For Markov chains this idea takes the form of hitting times: For 
A c S' we define the first hitting time of A as the random variable 

T^(w) := inf{n > 1, X„{u}) 6 A}, 

with the understanding that ta{oj) = oo if the inf does not exist. We often drop the 
variable w and denote the (conditional) distributions of the first hitting times by 

fn{i,A) := P{t_a = n\Xo = i}. 

Using this idea we can generalize the Chapman-Kolmogorov equation to random 
intermediate time points as follows (recall that we use the notation = I) 

n 

(5.2) PniiJ) = Y,fr{i,j)Pn-r{j,j) for n > 1. 

r=l 

In particular, the stochastic process with transition probability matrix P and initial 
variable ta is again a Markov chain for any A c S. 

The idea of a first hitting time leads to the definition of a sequence of random 
variables of subsequent visits to a state or set of states: Define := inf{n > 

Xn{uj) 6 A} for m > 1, with = ta{uj). With this notation we can 

write the number of visits of a set A c S' up to time m > 1 as 

m 

NA{m,U]) := Y,XA{Xn{ui)), 

n=l 

and the total number of visits as 

Na{u}) := Y,XA{Xn{u})) = #{m > 1, < oo}, 

n>l 

where XA denotes again the characteristic function of a set A. 

Two other concepts derived from first hitting times play a role in the analysis of 
Markov chains: The first one is the (conditional) probability of reaching a set of 
states A from a state i, i.e. 

P{u; € fl, A„(a;) 6 A for some n > 1 | Aq = f} = ^/„(i, A) =: F{i,A). 

n>l 

The other useful probabilistic concept is that of moments, where we will use only 
the first moment, i.e. the mean first hitting time 

KhA) := Y,nfn{i,A). 

n>l 

With these preparations we can define the ideas of recurrence and transience. When 
talking about points j e S we often use the notation /«(*, j), instead 

of etc. 
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Definition 5.3. A state z € S' is called recurrent, if P{ri < oo | Xq = i} = F{i,i) = 1. 
States that are not recurrent are called transient. 

A recurrent state i e S satisfying E(Ti | Xq = i) = /l(z,z) < oo is called positive 
recurrent, the other recurrent states are called null recurrent. Here we denote by 
E(y I Xq = i) the conditional expectation of a random variable Y under the measure 
P(-| Ao = z). 


We list some standard results regarding the classification of states in Markov chains, 
compare e.g., [inille] for the proofs. 

Theorem 5.4. Let be a Markov chain on the finite state space S. 

(1) A state j € S is recurrent iff 

T^PnijJ) = oo- 

neN 

(2) If the state j e S is recurrent then 

P{Aj- = oo I Ao = j} = 1. 

(3) A state j e S is transient iff 

T.PnijJ) < oo. 

neN 

(4) If the state j e S is transient then it holds for any i e S 

P{ Aj < oo I Xq = z} = 1 and 
E(Aj I Xq = z) < oo. 

(5) If the state j ^ S is recurrent and periodic of period 6 then 

lim PnsUJ) = / 

in particular, for aperiodic states we have 
lim Pn{j,j) = 


pUJ) 


(6) If the state j e S is positive recurrent and aperiodic, then for i e S arbitrary 
we have 


\\m.pn{i,j) = 


KjJ) ' 


For irreducible chains the qualitative behavior is uniform for all points, leading to 
the following results: 


Theorem 5.5. Let (A„)„<:n be an irreducible Markov chain on the finite state space 
S. We fix j € S, then for all i € S it holds that 

(1) If j has period 5, then so has i. 

(2) Iff is transient (recurrent, positive recurrent), then so is i. In fact, all 
states are either transient or positive recurrent. 

n 
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n+(5—1 

(5) If j is periodic of period 6 then \im pns(i,j) = , - and lim 4 Z Pk(i,j) = 

n-*-oo 7T,-»-oo^ 

particular for aperiodic states j we have limp„(i,j) = In 

all cases convergence is geometric with rate r < 1, where r = max{|A|, A 
is an eigenvalue of P with |A| < 1}, i.e. the ergodicity coefficient of the 
transition matrix P. 

For irreducible Markov chains the long term behavior for n ->• oo is described by er¬ 
godicity and stationarity. Both types of behavior can be formulated using invariant 
measures of the chain. 

Definition 5.6. Let be a Markov chain on the state space S. A probability 

distribution tt* on S is called invariant for the chain if tt* = T>{Xn) for all n 6 N. 
Here 2A(-) denotes again the distribution of a random variable. 

Remark 5.7. If is a Markov chain with transition probability matrix P on 

the hnite state space S = then the distribution tt* on S is invariant iff 

TT* ~ (tt^, k = l,...,d) € satisfies tt*'^ = 7r*^P, i.e. if tt* is a left eigenvector of 
P corresponding to the (real) eigenvalue 1. For irreducible chains this eigenvalue 
is a simple root of the characteristic polynomial of P, and it is the only one with 
absolute value equal to 1. 

Remark 5.8. Note that a Markov chain is (strictly) stationary (i.e., all its 

finite-dimensional distributions are invariant under time shift) iff its initial variable 
Xq has distribution 'D(Xo) = tt* for some invariant distribution tt*. 

Theorem 5.9. Let (A„)„(:n be an irreducible Markov chain on the finite state space 
S with transition probability matrix P. 

(1) The Markov chain has a unigue invariant distribution tt* on S . 

(2) The invariant distribution tt* (tt^, k = l,...,d) e satisfies 

where ii{k,k) denotes again the mean first return (hitting) time from k to 

k. 

(3) In particular, all states are positive recurrent. 

5.2. Markov Chains, Graphs, and Semiflows. Usually, presentations about 
finite state Markov chains develop the theory for irreducible chains, i.e. the chain 
consists of one communicating class. According to Theorems 15.51 and 15.91 the 
states of an irreducible chain behave uniformly in their limit behavior and thus no 
coexistence of transient and (positive) recurrent states is possible. We are interested 
in studying the qualitative behavior of general finite state Markov chains. Chapter 
9.8 of [15] presents some results from a matrix point-of-view. We will do so by 
using the results from Section 3. In this section, we develop the mechanisms that 
allow us to translate many of those results to the context of Markov chains. 

Let (A„)„(:n be a Markov chain on the finite state space S with transition proba¬ 
bility matrix P. We associate with P a weighted graph G = {V, E, w), where V = S, 
(i,j) € U iff p(i,j) > 0, and w ■■ E ^ [OJ] defined by w{{i,j)) = p{i,j). The 
adjacency matrix Aq is defined as in Definition 14 .1 1 for the graph G = {V,E). Note 
that the graph G = {V,E) is automatically an L-graph. Vice versa, let G = {V,E) 
be a graph with a weight function w '■ E ^ M. that satisfies the properties (i) 
w ■■ E ^ [Ojl] and (ii) X w((i,j)) = 1 for all i e V, then G = (V,E,w) can be 
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identified with the probability transition matrix P of a Markov chain on the state 
spaced via 

With this construction, we can try to interpret all concepts and results from Section 
I2.1l in the context of Markov chains. Basically, the key ingredient in all proofs is the 
following simple observation: Let 7 = {fp, *n) be a path in G = (V,E), then the 
probability P( 7 ) that this path occurs as a (finite length) trajectory of the chain 
with transition probability matrix P and initial distribution ttq is given 
by the joint probability 

(5.3) P( 7 ) = P{Xo = io,Xi = ii, ...,Xn-l = in-l,Xn = in] 

— Tq (f o)p(f 0 : ) * * * P(^n —2 ? f n —1 )p(f n—15 f n) • 

In particular, we obtain for any finite sequence io, of vertices in G: {io, 
is a path in G iff p{ia,ia+i) > 0 for a = 0,..., n - 1. 

This observation implies a probabilistic argument that we will need for our results 
in the next section. It is closely related to the no-cycle property of points for 
semiflows, compare Definition l3.10l 

Lemma 5.10. Let P be the transition probability matrix of a Markov chain on the 
state space S, and let G = (V,E) be its associated graph. Consider a set B cV and 
a point i e B with the properties 

(1) There exists a path 7 6 T" with 719 ( 7 ) = * =■ k ^ B for some n > I, 

(2) C>+(fc)nP = 0 . 

Then limp„(i, j) = 0 for all j ^ B uniformly at a geometric rate. 

Proof. Denote P( 7 ) = p, then by (j5.3l) we have Pn{i, k) > p and hence 

'ZPniiJ) < 1 -p- 

jsB 

Using the Chapman-Kolmogorov equation we compute for all j € B 
P 2 u{i,j) = 

ItS 

= T,P^(T0Pnil,j) + Puii,k)pn{k,j) 
li=k 

i Y.Pn{i,l)YlPnil,j)+0 

l^k l^k 

li=k 

<(i-p)^ 

Repeating this argument we obtain for all m > I 

Pmniij j) - ~ p) ■ 

By assumption 2 of the lemma, we see that Pmn+a{hj) ^ (1 - p)™ for 0 < a < 
n - 1, which proves the assertion. Note that the argument above even shows 
lim T, Pn{i,j) = 0 at the geometric rate (I - p). □ 

Next we comment briefly on the connection between Markov chains and semiflows 
defined by products of matrices, i.e. linear iterated function systems. The 
standard connection is constructed as follows: Let (W„)„ 5 P^ be a Markov chain on 
the finite state space 5" = {I, ...d} with transition probability matrix P. We identify 
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the probability measures on S with the set of probability vectors in W^, defined 
as V = {u € Ui > 0 for all i = and = !}• Then T : N x V V, 

defined by T(n,w) = is a semiflow. If uo = 'D{Xo), this semiflow describes 

the evolution of the 1-dimensional (and the n-dimensional) distributions of the 
Markov chain, thanks to the Chapman-Kolmogorov equation. Vice versa, if T is 
a linear, iterated function system on that leaves V invariant, then T can be 
interpreted as a Markov chain on the state space S' = {1,..., d}. In standard texts, 
many results about finite state Markov chains are proved using this connection. 
Our discussion in Section|4]suggests another matrix semiflow associated to a Markov 
chain, namely the semiflow 'k : N x ^ Q'^, 'I'(n,q) = {q^ ■* on the unit 

cube Q‘^ c R'^, compare Definition 14.81 Here Ae M(d, {0,1}) is defined by Uij = 1 if 
Pij > 0, and ttij = 0 otherwise. This semiflow does not propagate the distributions of 
the Markov chain, just its reachability or {0, l}-structure. It follows from Section 
|4] that all concepts and results for L-graphs can be interpreted in terms of this 
semiflow, e.g. Equation (15.31) and Lemma 15.101 have obvious translations to the 
context of 

Finally we consider semiflows on power sets: Let be a Markov chain on the 

finite state space S = {1, ...d} with transition probability matrix P. Let P(S') be the 
power set of S, define the semiflow $ : N x V{S) ViS), through <i>(n, A) = {j e 5, 
there exists i e A such that Pn{hj) > 0}. The flow $ is of the type (13.11) and hence 
it satisfies all the properties studied in Sections 13.1113.31 In particular. Equation 
(15.31) and Lemma 15. 101 have obvious translations to the context of $. 

Note that, in contrast to the semiflow T (and the weighted graph {V,E,w)), the 
semiflows and $ (and the graph G = {V,E)) do not, by definition, carry all the 
statistical (or distributional) information of the Markov chain (V„)„<:n, just the 
information about certain basic events (such as paths) occurring with probability 
zero or with positive probability. Hence when using this graph or these semiflows 
to analyze the Markov chain, we can only hope for some statements of the kind 
” an event defined by the Markov chain has positive probability or probability 0”. 
We will see in the next section that the graph and the semiflows do characterize a 
surprisingly wide array of properties of the Markov chain. 

5.3. Characterization of Markov Chains via Graphs and Semiflows. In 

this section let (V„)„(:n be a Markov chain on the finite state space S = {l,...d} 
with transition probability matrix P. When we talk about probability measures on 
S we always think of S as endowed with the discrete cr-algebra V{S). Associated 
with (V„)„<=N are the graph G = (S', E) and the semiflows $ : N xV(S) -*■ V{S) and 
4' : N X ^ as defined in the previous section. The goal of this section is to 
develop a “three-column dictionary” of equivalent objects and results in the three 
languages of Markov chains, graphs, and semiflows. To avoid confusion between 
the semiflows T : N x V V, defined on the set of probability vectors in R'^ and 
containing all the probabilistic information of the chain, and Jk : N x 
defined on the unit cube in R"* and containing only the reachability information 
of the chain, we will formulate our observations in terms of the semiflow $. The 
results immediately carry over to 4' using the correspondence from Section |4l 

5.3.1. Paths, Orbits, Supports of Transition Probabilities, and First Hitting Times. 
For a probability measure p. on {S,V{S)) the support supp/x is defined as the small¬ 
est subset S' c S such that p{S') = 1. Note that the n-step transition probabilities 
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Pni',-) define probability measures P{n,i,-) on S via P{n,i,A) := Y,j<iAPn{i,j)- 
That is, each row of the matrix P" “is” a probability measure for all n € N. 


Fact 1: A finite sequence of points (io, ■■■An) in S' is a path of G iff Pn(*o,*n) > 0 
iff in e $(n, {*o})- Each of these statements is equivalent to in 6 suppP(n, zq, •). 
The proof of this fact follows directly from (15.3|) . An immediate consequence of 
this fact is: 


Fact 2: Fix a point i e S. Then for j e S we have: j e 0^{i) iff j e suppP(n,i,-) 
for some n > 1 iff j € {*})• 


Fact 3: A subset A c S is forward invariant for G iff A is stochastically closed for 
the Markov chain iff $(n, A) c A for all n > 1. 

The proof of this fact follows directly from Fact 2. This fact also allows us to 
characterize certain properties of first hitting times for Markov chains. 


Fact 4: Let i e S and A c S'. The first hitting time distribution {fn{i,A), n > 1) 
satisfies for any n e N: /„(i,A) > 0 iff there exists a path 7 e T” with 710 ( 7 ) = i, 
TTn{j) 6 A, and 71 ^( 7 ) i A for all m = 1. Furthermore, F{i,A) > 0 iff 

O^{i)nA + 0. 

This result follows immediately from (15.311 and Fact 2. The definition of the semiflow 
$ in Equation (13.111 allows directly for an equivalent statement in terms of $. Note 
that we did not define “first hitting times” for graphs or semifiows because this 
concept is hardly, if ever, used in graph theory. However, if we define for the graph 
G = {S,E) the notion ctaC*) •= inf{n > 1, there exists 7 6 T" with 710 ( 7 ) = * and 
TTnil) s A} as the first hitting time of a set Aa S for paths starting in z € S', then 
we obviously have from Fact 4 that aA{i) = n implies fnii^ A) > 0, but the converse 
is, in general, not true. 

As a final observation for this section, we note that Definition 15.21 of the period of 
a state ie S only uses the property of Pn(z, i) > 0. Hence according to Fact 1, this 
is really a pathwise property of the graph G: Consider the graph G = (S, P) and a 
vertex i e S, we define the period of i as r]{i) = gcd{n > 1, there exists a path 7 e T" 
with 710 ( 7 ) = * s-nd 71 ^( 7 ) = i}. Then, by Fact 1, we have 77(1) = S(i), the period 
of z as a state of the Markov chain. It seems, however, that periods of vertices are 
rarely, if ever, used in graph theory. 


5.3.2. Communication and Communicating Classes. We have defined communicat¬ 
ing classes for graphs in Definition 12.51 and for Markov chains after Definition 15.11 
These definitions differ slightly: For graphs we required paths of length n > 1, while 
we followed standard practice for Markov chains and allowed n = 0. The Markov 
chain practice renders the communication relation an equivalence relation, but 
it also leads to trivial communicating classes of the form {z} with p„(z,z) = 0 for 
all n > 1. In the context of graphs we called such vertices transitory, compare 
Definition 12.91 Hence we obtain the following results. 
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Fact 5: Consider two points i,j € S. i and j communicate via the graph G iff i 
and j communicate via the Markov chain and > 0 and > 0 for some 

n, m > 1. 

Fact 6: A point f 6 S' is a transitory vertex of the the graph G iff Pn{i,i) = 0 for 
all n > 1. 

Fact 7: A set A c S is weakly invariant for the semiflow $ iff for all n e N there 
exist in,jn 6 A with > 0. 


Fact 8: A subset G c S is a communicating class of the graph G iff G is a nontrivial 
communicating class of the Markov chain iff G is a finest Morse set of the semiflow 


Fact 9: A subset G c S is a maximal communicating class of the graph G iff G is 
a stochastically closed communicating class of the Markov chain iff G is a minimal 
(with respect to set inclusion) attractor of 


Fact 10: A point i & S satisfies i € G for some communicating class G of the graph 
G iff there exists a sequence Uk with pni^{i,i) > 0 iff z € a;(A), the w-limit set 
of some A c S' under the semiflow $ iff {z} is a recurrent point of 
As we will see below, recurrence for semiflows (as defined in Definition 13.9() and for 
Markov chains (as defined in Definition 15.31) are two different concepts, as are the 
concepts of transitory points of semiflows and transient points of Markov chains. 

5.3.3. Recurrence, Transience, and Invariant Measures for Markov Chains. Let 
M c S be a stochastically closed set for Markov chain (A„)„gN with transition prob¬ 
ability matrix P. The restriction of P to M defines a new Markov chain (A,^)„gN, 
whose transition probability matrix we denote by P^. Since M is stochastically 
closed, we have for all i,j e M that Pn{i,j) = Pni^J) holds for all n e N. Therefore 
we conclude from Theorem 15.41 f e M is a transient (recurrent, positive recurrent) 
point for X iff it is transient (recurrent, positive recurrent) for the restricted chain 
X^. And Theorems 15.51 and 15.91 imply that p is an invariant probability measure 
of X with suppp, n M + 0 iff /r is an invariant probability measure for X^. With 
these preparations we can characterize the long term behavior of states in a Markov 
chain: 


Fact 11: A point i e S' is a transient point of the Markov chain (A„)„ 5 n iff * e S\uC, 
C a maximal communicating class of the graph G = {S,E). 

Proof. Maximal communicating classes G of G are forward invariant by Remark 
12.261 Hence the Markov chain X^' is irreducible and all states i e C are (positive) 
recurrent by Theorem 15.91 3. From the observation above we then have that i is 
(positive) recurrent for X, which proves the => direction. Vice versa, if f 6 S\u G, 
then Lemma r5.101 with B = uG implies \impn{i,i) = 0 at a geometric rate. Hence 

n-»-oo _ 

Y, Pn{i,i) < °° and therefore the state i is transient by Theorem l5.41 3. □ 

n€N 
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Note that the transitory points for the graph G = (S', E) are a subset of the transient 
states of the Markov chain: Only those points are transitory that are not element 
of any communicating class. 


Fact 12: A point * € S is a recurrent (and positive recurrent) point of the Markov 
chain iff i 6 uC, C a maximal communicating class of the graph G = {S,E). 

The proof of this fact is immediate from Fact 11 and Theorems 15.41 and 15.91 3. 
Note that the semiflow concept of recurrence is different from that for Markov 
chains: Under the semiflow $ the points in any communicating class are exactly 
the ^-recurrent points. 

We now turn to invariant distributions of the Markov chain (A„)„<:n. If fj,i and 
^2 are two invariant probability measures of a Markov chain, then any convex 
combination /i = a/ri + (1 - a)/i ,2 for a e [0,1] is obviously again an invariant 
probability measure. An invariant probability measure /r is called extreme if it 
cannot be written as a convex combination /i = a/ii + (1 - a)/i 2 of two different 
invariant probability measures /ii + /i 2 for a € (0,1). 


Fact 13: Each maximal communicating class Gv of the graph G = {S,E) is the 
support of exactly one invariant distribution of the Markov chain (A'n)^^^. The 
chain has exactly I extreme invariant distributions where I is the number 

of maximal communicating classes of the graph G. Any invariant distribution fj, of 

i 

the chain is a convex combination fJ. = Y, with = 1- 

V=1 

Proof. Maximal communicating classes Ci, of G are stochastically closed, hence the 
Markov chain restricted to C^, is irreducible and therefore C^, is the support of a 
unique invariant distribution fXi, by Theorem 15.91 1. Now the other claims follow 
directly from Fact 9. □ 


Fact 14: The Markov chain has a unique (in the distributional sense) 

stationary solution (A)()„gN on each maximal communicating class Gi, of the graph 
G = {S,E), obtained by taking initial random variables Xq with distribution 
2?(Ao) = /rjy. All other stationary solutions are convex combinations of the X'', 
12 — 1 , ..., /. 

This fact is a combination of Remark 15.81 and Fact 13. 

5.3.4. Global Behavior and Multistability. This section is devoted to the conver¬ 
gence behavior of a Markov chain (X„)„gf} as n ^ oo. We have already charac¬ 
terized the transient and (positive) recurrent points. It remains to clarify how the 
chain behaves starting in any of these points. When one deals with difference (or 
differential) equations, one first looks for fixed points (and other simple limit sets) 
and then tries to find the initial values from which the system converges towards 
these fixed points (or more generally, limit sets). The study of the global behavior 
of dynamical systems clarifies these issues, compare, e.g., [4] for the case of flows, 
and Section 3 for an adaptation to specific semiflows related to L-graphs and to 
Markov chains. 
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Markov chains, in general, do not have fixed points. According to Fact 10, their long 
term behavior is determined by the communicating classes. And Fact 12 suggests 
that the maximal communicating classes determine the behavior of a Markov chain 
as n -»■ oo. The following facts show that this is, indeed, true. The special 
case of a fixed point is recovered when a maximal communicating class consists of 
exactly one state; such states are called absorbing. 


Fact 15: Consider the Markov chain with transition probability matrix 

P, initial random variable Xq and initial distribution 'D(Xo) = ttq. Let C be a 
maximal communicating class of the graph G = {S,E) with invariant probability 
^*. Assume that ttq is concentrated on C, i.e. 7ro(i) > 0 iff i € C. Then 


Um-'£v(Xk) = fi* 


n^oo ri 


k=l 


in the distributional sense, i.e. as vectors in K.'^. If, furthermore, C has period S 
then 

1 n+S-1 

hm - ^ V(Xk) = 

In particular for C aperiodic we have limP(X„) = fi*. All convergences are at a 

n->oo 

geometric rate. This fact follows immediately from Theorems 15.51 5 and 15.91 2. and 
the remarks at the beginning of Section 15.3.31 


Fact 16: Consider the Markov chain (A„)„gN- Let C = uC^, a maximal 
communicating class of the graph G = {S, E), and D := S\G. Pick i e D. Then the 
first hitting time tq satisfies P{tc < oo | Xq = z} = 1 and E(rc) < oo. 

Proof. Assume, under the given assumptions, that P{tc < oo | Xq = z} < 1, i.e. 
P{rc = oo I Xq = z} > 0. Then there exists a state j e S\G with P{Nj = oo | Xq = 
j} > 0. By the characterization from Theorem l5.4l 4 the state j cannot be transient, 
and hence by the dichotomy in Theorem 15.51 2 j is recurrent, which contradicts 
Fact 12. Hence P{tc < oo | Xq = z} = 1 and then E(tc) < oo follows from geometric 
convergence in Lemma 15.101 □ 

We define the probability that the chain, starting in P = S'\C', hits the maximal 
communicating class Gi, by Pi, := f‘{Xrc ^ Gi,\ Xq e D}. Note that Y.Pv = 1- Then 
the long-term behavior of with 'D{Xq) concentrated in D is as follows: A„ 

will leave the set D of transient states in finite time (even with finite expectation) 
and enter into the set G of (positive) recurrent points, where it may enter one 
or more of the depending on whether pi, is positive or not. By the random 
version of the Chapman-Kolmogorov equation (15.2L the process continues as a 
Markov chain, and hence in each Ci, follows the behavior described in Fact 15. In 
particular, if all Gu are aperiodic, we obtain: 


Fact 17: Consider the Markov chain {Xn)nm with maximal communicating classes 
Cl,..., Gi and extreme invariant measures /ii,..., pi. Assume that all Gu, 1^=1 ,..., I 
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are aperiodic. Then we have as limit behavior of the chain 

i 

n->oo 

1^=1 

where the Pi, are as defined above. 

This fact leads to the definition of multistable states: A state i e S is called multi¬ 
stable for the Markov chain if there exist maximal communicating classes 

Cl and C 2 with Ci + C 2 such that pi, > 0 for 1 / = 1,2. We now obtain the following 
fact as a criterion for the existence of multistable states. 

Fact 18: Let {Xn)nm be a Markov chain with connected graph G. Then the chain 
has multistable states iff G has at least two maximal communicating classes. 
Multistable, and specifically bistable states play an important role in many appli¬ 
cations of stochastic processes in the natural sciences and in engineering see [8]. 
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